Image-based cell sorting systems and methods

ABSTRACT

Disclosed are systems, devices and methods for imaging and image-based sorting of particles in a flow system, including cells in a flow cytometer. In some aspects, a system includes a particle flow device to flow particles through a channel, an imaging system to obtain image data of a particle during flow through the channel, a processing unit to determine a property associated with the particle and to produce a control command for sorting the particle based on sorting criteria associated with particle properties, and an actuator to direct the particle into one of a plurality of output paths of the particle flow device in real-time.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent document claims priorities to and benefits of U.S.Provisional Patent Application No. 62/348,511 entitled “FLOW CYTOMETERWITH IMAGE-BASED CELL SORTING” filed on Jun. 10, 2016. The entirecontent of the aforementioned patent application is incorporated byreference as part of the disclosure of this patent document.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under grant no.1R43DA042636-01 awarded by the National Institutes of Health (NIH). Thegovernment has certain rights in the invention.

TECHNICAL FIELD

This patent document relates to systems, devices and techniques forparticle sorting in fluid, including flow cytometry devices andtechniques and applications in chemical or biological testing anddiagnostic measurements.

BACKGROUND

Flow cytometry is a technique to detect and analyze particles, such asliving cells, as they flow through a fluid. For example, a flowcytometer device can be used to characterize physical and biochemicalproperties of cells and/or biochemical molecules or molecule clustersbased on their optical, electrical, acoustic, and/or magnetic responsesas they are interrogated by in a serial manner. Typically, flowcytometry use an external light source to interrogate the particles,from which optical signals are detected caused by one or moreinteractions between the input light and the particles, such as forwardscattering, side scattering, and fluorescence. Properties measured byflow cytometry include a particle's relative size, granularity, and/orfluorescence intensity.

Cell sorting, including cell sorting at the single-cell level, hasbecome an important feature in the field of flow cytometry asresearchers and clinicians become more interested in studying andpurifying certain cells, e.g., such as stem cells, circulating tumorcells, and rare bacteria species. Cell sorting can be achieved byvarious techniques.

Flow cytometry devices and systems can be implemented based onmicrofluidic technologies for research assays and diagnostics as well asfor clinical applications. A microfluidic device is an instrument thatcan control the behavior of very small amounts of fluid (e.g., such asnL, pL, and fL) through channels with dimensions in relatively smalldimensions, e.g., the sub-millimeter range. Microfluidic devices can beimplemented to obtain a variety of analytical measurements includingmolecular diffusion values, chemical binding coefficients, pH values,fluid viscosity, molecular reaction kinetics, etc. Microfluidic devicescan be built on microchips to detect, separate and analyze biologicalsamples, which can also be referred to as a lab-on-a-chip. For example,a microfluidic device may use biological fluids or solutions containingcells or cell parts to diagnose diseases. Inside microfluidic channelsof, for example, a microfluidic flow cytometer, particles includingcells, beads, and macromolecules can be interrogated according to theiroptical, electrical, acoustic, and/or magnetic responses using flowcytometry techniques.

SUMMARY

The technology disclosed in this patent document can be implemented toprovide methods, devices and systems for producing images of particlesin a flow system, and in specific configurations, the disclosedtechnology can be used for imaging particles in real time andsubsequently sorting particles, including cells, based on the spatialinformation from the image. The disclosed techniques can be applied forproducing cell images and sorting cells in flow cytometers. Inapplications, the disclosed technology can be used to detect and sortcells based on the fluorescent and/or scattering intensity by takinginto account the spatial information such as the spatial distribution offluorescence.

In implementations, for example, the disclosed systems possess the highthroughput of flow cytometers and high spatial resolution of imagingcytometers, in which the cell images are produced at a fast enough rateto accommodate real-time cell sorting in a flow system based on physicaland/or physiological properties of the cell, e.g., as opposed to just adetection event.

In some aspects, an image-based particle sorting system includes aparticle flow device structured to include a substrate, a channel formedon the substrate operable to flow cells along a flow direction to afirst region of the channel, and two or more output paths branching fromthe channel at a second region proximate to the first region in thechannel; an imaging system interfaced with the particle flow device andoperable to obtain image data associated with a cell when the cell is inthe first region during flow through the channel; a data processing andcontrol unit in communication with the imaging system, the dataprocessing and control unit including a processor configured to processthe image data obtained by the imaging system to determine one or moreproperties associated with the cell from the processed image data and toproduce a control command based on a comparison of the determined one ormore properties with a sorting criteria, in which the control command isproduced during the cell flowing in the channel and is indicative of asorting decision determined based on one or more cellular attributesascertained from the image signal data that corresponds to the cell; andan actuator operatively coupled to the particle flow device and incommunication with the actuator, the actuator operable to direct thecell into an output path of the two or more output paths based on to thecontrol command, in which the system is operable to sort each of thecells during flow in the channel within a time frame of 15 ms or lessfrom a first time of image capture by the imaging system to a secondtime of particle direction by the actuator.

In some aspects, a method for image-based particle sorting includesobtaining image signal data of a cell flowing through a channel of aparticle flow device; processing the image signal data to produce animage data set representative of an image of the cell; analyzing theproduced image data set to identify one or more properties of the cellfrom the processed image data; evaluating the one or more identifiedproperties of the cell with a sorting criteria to produce a controlcommand to sort the cell based on one or more cellular attributesascertained from the image signal data corresponding to the cell duringcell flow in the particle flow device; and directing the cell into oneof a plurality of output paths of the particle flow device based on tothe control command.

In some aspects, an image-based particle sorting system includes aparticle flow device structured to include a substrate, a channel formedon the substrate operable to flow particles along a flow direction to afirst region of the channel, and two or more output paths branching fromthe channel at a second region proximate to the first region in thechannel; an imaging system interfaced with the particle flow device andoperable to obtain image data associated with a particle when theparticle is in the first region during flow through the channel; a dataprocessing and control unit in communication with the imaging system,the data processing and control unit including a processor configured toprocess the image data obtained by the imaging system to determine oneor more properties associated with the particle from the processed imagedata and to produce a control command based on a comparison of thedetermined one or more properties with a sorting criteria; and anactuator operatively coupled to the particle flow device and incommunication with the actuator, the actuator operable to direct theparticle into an output path of the two or more output paths based on tothe control command, in which the system is operable to sort each of theparticles during flow in the channel within a time frame of 15 ms orless from a first time of image capture by the imaging system to asecond time of particle direction by the actuator.

In some aspects, a method for image-based sorting of a particle includesobtaining image signal data of a particle flowing through a channel of aparticle flow device; processing the image signal data to produce animage data set representative of an image of the particle; analyzing theproduced image data set to identify one or more properties of theparticle from the processed image data; producing a control command byevaluating the one or more identified properties with a sortingcriteria; and directing the particle into one of a plurality of outputpaths of the particle flow device based on to the control command.

The above and other aspects of the disclosed technology and theirimplementations and applications are described in greater detail in thedrawings, the description and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows a diagram of an example embodiment of an image-basedparticle sorting system in accordance with the disclosed technology.

FIG. 1B shows a block diagram of an example embodiment of a dataprocessing and control unit of an image-based particle sorting system inaccordance with the disclosed technology.

FIGS. 2A-2C show diagrams of an example embodiment of an image-basedcell sorting microfluidic system in accordance with the disclosedtechnology.

FIGS. 3A and 3B show example captured and processed image data from PMTsignals to produce cell images for image-based particle sortingtechniques in accordance with the disclosed technology.

FIGS. 4A-4C show diagrams of example embodiments of processes ofimage-based particle sorting methods in accordance with the presenttechnology.

FIG. 4D shows a brightness-time plot depicting an example cell detectionprocess implementation based on the processing of a time-domain PMTsignal associated with a single cell traveling through the illuminationarea of an example system in accordance with the present technology.

FIG. 5A shows a diagram of an example implementation of a process todetermine an identified feature or features of a detected cell forimage-based cell sorting.

FIG. 5B shows a data plot depicting the results of an implementation ofan example image reconstruction technique in accordance with the presenttechnology.

FIG. 6. shows a data plot depicting the fluorescence intensity signalfor a cell based on an example image reconstruction.

FIG. 7 shows an example of a reconstructed image of a cell.

FIGS. 8A and 8B show example fluorescence microscope images oftransfected but not drug treated cells and transfected and drug treatedcells.

FIGS. 9A and 9B show example fluorescence cell images taken by theimage-based cell sorter system of transfected but not drug treated cellsand transfected and drug treated cells.

FIG. 10 shows a histogram of the example calculated fluorescence area ofall events and of the sorted cells from an example implementation.

FIG. 11 shows a diagram of an example embodiment of an image-based cellsorting microfluidic system in accordance with the disclosed technology.

FIG. 12 shows a flow diagram of an example image processingimplementation using the system shown in FIG. 11.

FIG. 13A shows a flow diagram depicting example data through the exampleimage processing steps shown in FIG. 12.

FIG. 13B shows an example distribution plot for subsets using an exampleReceiver Operating Characteristics (ROC) technique to evaluate extractedparameters such as cell morphology parameters.

FIGS. 14A and 14B show cell images of un-translocated cells andtranslocated cells, respectively, captured by the example system andreconstructed by example image reconstruction techniques in accordancewith the disclosed technology.

FIG. 15 shows an example of a hyperplane exhibiting separation of thetwo cell sets for implementations of cell sorting criteria.

FIGS. 16A and 16B show cell images of un-translocated cells andtranslocated cells, respectively, captured by the example system, via amicroscope.

FIGS. 17A and 17B show cell images of G1 phase cells and G2/M phasecells, respectively, captured by the example system and reconstructed byexample image reconstruction techniques in accordance with the disclosedtechnology.

FIG. 18 shows an example of a hyperplane exhibiting separation of thetwo cell sets for implementations of cell sorting criteria.

FIGS. 19A and 19B show cell images of G1 phase cells and G2/M cells,respectively, captured by the example system and reconstructed byexample image reconstruction techniques in accordance with the disclosedtechnology.

FIG. 20 shows a flow diagram of an example image processingimplementation for sorting based on number of beads bonded on the cellmembrane of cells implemented by the system shown in FIG. 11.

FIG. 21 shows examples of grayscale cell images processed by an exampleimage processing module with different number of beads.

FIG. 22 shows a histogram of beads image area for different number ofbeads.

DETAILED DESCRIPTION

Some existing flow cytometer devices and systems detect and sort cellsbased on the fluorescence and/or scattering intensity without takinginto account the spatial information such as the spatial distribution offluorescence. There has been some advancements in the development oftechniques to produce cell images for flow cytometers with highthroughput and high spatial resolution. However, the cell image has notbeen produced in a fast enough rate to be useful for applications, inparticular for cell sorting in a flow system, e.g., due to the requiredamount of computation to generate the cell image. As such, existingstate of the art for cell sorting capabilities are “detection only”systems, and fail to “screen” the detected cells based on meaningful andnuanced criteria.

In applications, the disclosed technology can be implemented in specificways in the form of methods, systems and devices for image-based cellsorting in flow cytometry using (a) real-time image acquisition of fasttravelling cells by efficient data processing techniques utilizingmathematical algorithms implemented with FPGA and/or GPU and concurrent(b) “gating” techniques based on spatial characteristics of theparticles as the sorting criteria from the real-time acquired images.Unlike traditional flow cytometers that use fluorescent intensities ofchosen biomarkers as criteria for cell sorting, the methods, systems anddevices in accordance with the disclosed technology allow for varioususer-defined gating criteria containing spatial features.

Examples of image-based gating criteria include cell contour, cell size,cell shape, size and shape of internal cell structures such as the cellnucleus, fluorescent patterns, fluorescent color distribution, etc. Forexample, users can draw the cells they wish to separate and the systemwill perform accordingly. With such unique capabilities, users such asresearchers can track many important biological processes bylocalization of certain proteins within cytosolic, nuclear, or cellmembrane domains and subdomains. Because every cell population has somedegree of heterogeneity at a genomic (e.g., mutations, epigenetics) orenvironmental (e.g., asymmetric division, morphogen gradients) level,identification and extraction of single-cells according to their uniquespatial features are envisioned to contribute significantly to thefields of immunology, tumor heterogeneity, stem cell differentiation,and analysis of neurons.

In some embodiments, an image-based particle sorting system includes aparticle flow device, such as a flow cell or a microfluidic device,integrated with a particle sorting actuator; a high-speed andhigh-sensitivity optical imaging system; and a real-time cell imageprocessing and sorting control electronic system. For example, anobjective of the disclosed methods, systems and devices is to performthe entire process of (i) image capture of a particle (e.g., cell), (ii)image feature reconstruction from a time-domain signal, and (iii) makinga particle sorting decision and sorting operation by the actuator withina latency of less than 15 ms to fulfill the needs for real-time particlesorting. In some implementations described herein, the total latency isless than 8 ms (e.g., 7.8 ms), in some implementations, the totallatency is less than 6 ms (e.g., 5.8 ms), and in some implementations,the total latency is less than 3.5 ms (e.g., 3.3 ms). Forimplementations of cell sorting, for examples, the disclosed methods,systems and devices are able to image, analyze and sort cells by imagefeatures specific to life cycles, protein localization, genelocalization, DNA damages, and other cellular properties, which can beconnected to different diseases or pathogens.

FIG. 1A shows a diagram of an example embodiment of an image-basedparticle sorting system 100 in accordance with the present technology.The system 100 includes a particle flow device 110, an imaging system120 interfaced with the particle flow device 110, a data processing andcontrol unit 130 in communication with the imaging system 120, and anactuator 140 in communication with the data processing and control unit130 and operatively coupled to the particle flow device 110. Theparticle flow device 110 is structured to include a channel 111 in whichparticles flow along a flow direction to an interrogation area 115 whereimage data are obtained by the imaging system 120 for each particle inthe interrogation area 115. The data processing and control unit 130 isconfigured to process the image data and determine one or moreproperties associated with the particle to produce a control command forsorting of the particle. The control command is provided to the actuator140, which is interfaced with the particle flow device 110 at a sortingarea of the device 110, such that the actuator operates to sort theparticular particle into an output channel corresponding to the controlcommand. The system 100 implements image-based sorting of the particlesin real-time, in which a particle is imaged by the imaging system 120 inthe interrogation area and sorted by the actuator 140 in the sortingarea in real time and based on a determined property analyzed by thedata processing and control unit 130.

The system 100 is user-programmable to sort each particle based onuser-defined criteria that can be associated with one or more of aplurality of properties exhibited by each individual particle analyzedin real time by the data processing and control unit 130. Some exampleuser-defined criteria include, but are not limited to, an amount and/orsize of sub-features of or on the individual particle (e.g.,sub-particles attached to living cells, including particles engulfed bycells or attached to cells); morphology of the individual particle;and/or size of the individual particle. In this manner, the system 100is able to evaluate and sort particles by properties, such as propertiesof living cells, including sorting by cellular physiologicalfunctionalities (e.g., particle or substance uptake by a cell, orparticle engulfment by a cell), by cell damage, by localization ofproteins, or by other cellular properties.

FIG. 1B shows a block diagram of an example embodiment of the dataprocessing and control unit 130. In various implementations, the dataprocessing and control unit 130 is embodied on one or more personalcomputing devices, e.g., including a desktop or laptop computer, one ormore computing devices in a computer system or communication networkaccessible via the Internet (referred to as “the cloud”) includingservers and/or databases in the cloud, and/or one or more mobilecomputing devices, such as a smartphone, tablet, or wearable computerdevice including a smartwatch or smartglasses. The data processing andcontrol unit 130 includes a processor 131 to process data, and memory132 in communication with the processor 131 to store and/or buffer data.For example, the processor 131 can include a central processing unit(CPU) or a microcontroller unit (MCU). In some implementations, theprocess 131 can include a field-programmable gate-array (FPGA) or agraphics processing unit (GPU). For example, the memory 132 can includeand store processor-executable code, which when executed by theprocessor 131, configures the data processing and control unit 130 toperform various operations, e.g., such as receiving information,commands, and/or data, processing information and data, such as from theimaging system 120, and transmitting or providing processedinformation/data to another device, such as the actuator 140. To supportvarious functions of the data processing and control unit 130, thememory 132 can store information and data, such as instructions,software, values, images, and other data processed or referenced by theprocessor 131. For example, various types of Random Access Memory (RAM)devices, Read Only Memory (ROM) devices, Flash Memory devices, and othersuitable storage media can be used to implement storage functions of thememory 132. In some implementations, the data processing and controlunit 130 includes an input/output (I/O) unit 133 to interface theprocessor 131 and/or memory 132 to other modules, units or devices. Insome embodiments, such as for mobile computing devices, the dataprocessing and control unit 130 includes a wireless communications unit,e.g., such as a transmitter (Tx) or a transmitter/receiver (Tx/Rx) unit.For example, in such embodiments, the I/O unit 133 can interface theprocessor 131 and memory 132 with the wireless communications unit,e.g., to utilize various types of wireless interfaces compatible withtypical data communication standards, which can be used incommunications of the data processing and control unit 130 with otherdevices, e.g., such as between the one or more computers in the cloudand the user device. The data communication standards include, but arenot limited to, Bluetooth, Bluetooth low energy (BLE), Zigbee, IEEE802.11, Wireless Local Area Network (WLAN), Wireless Personal AreaNetwork (WPAN), Wireless Wide Area Network (WWAN), WiMAX, IEEE 802.16(Worldwide Interoperability for Microwave Access (WiMAX)), 3G/4G/LTEcellular communication methods, and parallel interfaces. In someimplementations, the data processing and control unit 130 can interfacewith other devices using a wired connection via the I/O unit 133. Thedata processing and control unit 130 can also interface with otherexternal interfaces, sources of data storage, and/or visual or audiodisplay devices, etc. to retrieve and transfer data and information thatcan be processed by the processor 131, stored in the memory 132, orexhibited on an output unit of a display device or an external device.

FIGS. 2A-2C show diagrams of an image-based cell sorting microfluidicsystem 200 in accordance with some embodiments of the image-basedparticle sorting system 100. The system 200 includes a microfluidicdevice 210 to flow particles through an optical interrogation channelfor sorting, an imaging system 220 to obtain image data of the particlesin an illumination area of the interrogation channel, a data processingand control system 230 to process the obtained image data in real timeand determine a sorting command, and an actuator 240 to gate theparticles in the microfluidic device 210 based on the determined sortingcommand.

As shown in FIG. 2A and FIG. 2C, the microfluidic device 210 structuredto include a substrate 213 having a passage forming a microfluidicsample channel 211, and microfluidic sheath channels 212 that convergeupon the sample channel 211. In implementations, for example, the samplechannel is configured to carry particles (e.g., cells) suspended in afluid that flows in a flow direction, and the sheath channels 212 areconfigured to provide sheath flow of fluid to hydrodynamically focus thesuspended particles in the fluid prior to flowing through anillumination area 215 of the microfluidic device 210. In someembodiments, for example, the substrate 213 can be formed in a bulkmaterial, e.g., such as polydimethylsiloxane (PDMS), that is bonded to abase substrate, e.g., a glass base substrate or base substrate of othermaterial.

The imaging system 220 of the system 200 includes a light source 221,e.g., a laser, to provide an input or probe light at the illuminationarea 215 of the microfluidic device 210, and an optical imager 222 toobtain images of the illuminated particles in the illumination area 215.The example optical imager 222, as shown in FIG. 2A, includes anobjective lens 223 (e.g., of a microscope or other optical imagingdevice) optically coupled to a spatial filter (SF) 224, an emissionfilter (EF) 225, and a photomultiplier tube (PMT) 226. In someimplementations, for example, the imaging system 220 includes one ormore light guide elements 229 to direct the input light at theillumination area 215 of the microfluidic device 210. In the exampleshown in FIG. 2A, the light guide element 229 includes a dichroic mirrorarranged with the light source 221 and the optical imager 222 to directthe input light at the illumination area 215.

In some implementations of the imaging system 220, the light source 221(e.g., the laser) is configured to produce a fluorescent excitationsignal that is incident upon the illumination area 215 to cause afluorescent emission by the particles. The optical imager 222 capturesthe optical output fluorescent emission signal such that an image of theparticle can be generated.

The data processing and control system 230 of the system 200 isconfigured in communication with the optical imager 222, e.g., via thePMT, to rapidly process the imaged particles and produce a sortingcontrol based on the processed image of each particle imaged inreal-time. In some implementations of the data processing and controlunit 230, a FPGA processing unit is configured to rapidly process theimage signal data received by the optical imager 222. An example of suchimplementations can include a Virtex-II (xc2v3000) FPGA platform inconjunction with a compiler Xilinx 10.1, which can be provided via achassis Crio-9104 from National Instrument to execute algorithms inaccordance with data processing methods of the present technology.

The actuator 240 of the system 200 is configured in communication withthe real-time data processing and control system 230 to gate theparticle flowing in a gate area 217 of the sample channel 211 into twoor more output channels 218 of the microfluidic device. In someembodiments, for example, the distance between the illumination area 215and the gate area 217 can be in a range of 50 μm to 1 mm. Inimplementations, the actuator 240 receives the sorting command from thedata processing and control system 230 in real time, such that theimaging system 220 and data processing and control system 230 operate tocapture and process the image of each particle while flowing through theillumination area 215 so that the actuator 240 receives and executes thesorting command to gate each particle accordingly. For example, in someimplementations, the actuator 240 includes a piezoelectric actuatorcoupled to the substrate 213 to produce deflection that causes theparticle to move in a particle direction in the gate area 217 thatdirects the particle along a trajectory to one of the two or more ofoutput channels 218.

In implementations of the system 200, for example, the suspended singlecells are hydrodynamically focused in the microfluidic channel by sheathflow, ensuring that the cells travel in the center of the fluidicchannel at a uniform velocity. A fluorescence emission is detected bythe PMT 226 in a wide-field fluorescence microscope configuration, suchas in the example shown in FIG. 2A. In this example, to accommodate thegeometry of the microfluidic device, the laser beam is introduced to theoptical interrogation by 52-degree reflection by a miniature dichroicmirror (DM) 229 positioned in front of a 50× objective lens 223 (e.g.,with NA=0.55, working distance=13 mm).

FIG. 2B shows an example embodiment of the spatial filter (SF) 225 thatis inserted in the detection path right at the image plane of theoptical imager 222 of the imaging system 220. The spatial filter designincludes a pattern having a plurality of slits positioned apart. In someembodiments, for example, the spatial filter 225 includes a pattern ofopenings having uniform dimensions, in which the pattern of openingsencodes a waveform on the received light by the optical imager. In theexample shown in FIG. 2B, the pattern includes ten 100 μm by 50 μm slitspositioned apart in the way of one is immediately after another. In someembodiments, for example, the spatial filter 225 includes a pattern ofopenings having a varying longitudinal and transverse dimensions withrespect to the flow direction across the microfluidic channel, such thata waveform is encoded by the optical imager to allow optical detectionof a position of a particle in at least two dimensions in theillumination area 215 of the microfluidic channel 211. An example oftwo-dimensional spatially-varying spatial filter is provided in U.S.Pat. No. 9,074,978 B2 entitled “OPTICAL SPACE-TIME CODING TECHNIQUE INMICROFLUIDIC DEVICES”, the entire content of which is incorporated byreference as part of this disclosure for all purposes.

Referring back to FIG. 2A, although the imaging system 220 shows onlyone PMT for detection of fluorescent signal, it is understood that morePMTs can be added to the optical imager 222, and, if necessary, moreexcitation laser beams added to the light source 221, to producemulti-color fluorescent cell images.

The real-time data processing and control system 230 includes a controlloop system that is implemented using a field-programmable gate-array(FPGA) to process the captured images and produce the correspondingsorting commands for each particle. The data processing and controlsystem 230 includes image processing and image-based particle sortingalgorithms executable by the FPGA to provide automated cell imagegeneration and accurate sorting by the system 200. For example, once asorting decision is made by the FPGA algorithm, the example on-chipintegrated piezoelectric lead-zirconate-titanate (PZT) actuator 240 isactuated to apply fluidic pressure at the nozzle structure at thesorting junction. In some implementations, the example PZT actuator 240is configured to execute the fluid displacement operation in response tothe sorting command in a time frame at or less than 0.1 ms peroperation. For example, the fluid displacement by the PZT actuator inless than 0.1 ms exhibits single cell hydrodynamic manipulationcapabilities with a high throughput.

Other examples of features of a particle flow device and/or an actuatorthat can be used in example embodiments of the devices, systems, andmethods in accordance with the disclosed technology are provided in U.S.Pat. No. 9,134,221 B2 entitled “FLUIDIC FLOW CYTOMETRY DEVICES ANDPARTICLE SENSING BASED ON SIGNAL ENCODING”, the entire content of whichis incorporated by reference as part of this disclosure for allpurposes. Other examples of features of an optical imaging system thatcan be used in example embodiments of the devices, systems, and methodsin accordance are provided in U.S. patent application Ser. No.15/514,930 which is a U.S. National Stage Application filing under 35U.S.C. § 371 based on PCT Application No. PCT/US2015/053368 entitled“IMAGING FLOW CYTOMETRY USING SPATIAL-TEMPORAL TRANSFORMATION” andpublished as WO2016/054293A1, the entire content of which isincorporated by reference as part of this disclosure for all purposes.

FIGS. 3A and 3B show example results of a PMT signal and thefluorescence cell image constructed from the PMT signal, processed bythe example data processing and control system 230, using an examplealgorithm described in FIG. 4 and executed by the example FPGA. FIG. 3Ashows an example time-domain PMT output signal of fluorescent light froma A549 cell stained with CellTrace CF SE. FIG. 3B shows thecorresponding processed (e.g., re-sized) fluorescence image depictingthe real size of the cell. The numbered regions segmented by dashedlines in the figures demonstrate the correspondence between thetime-domain signal and the resulting image. Size is labeled in FIG. 3B.

The spatial resolution of the restored image in x- (transverse)direction depends on the number of the slits on the spatial filter, andin y- (cell-travelling) direction depends on the sampling rate and cellflow rate. In the original image restored by the imaging flow cytometer(shown in FIG. 3B), the effective pixel size is 2 μm in x-direction andabout 0.4 μm in y-direction. The recovered image represents a 20 μm by20 μm area in the object plane in the microfluidic channel.

FIG. 4A shows a diagram of an example embodiment of a method 400 forimage-based sorting of particles. Implementations of the method 400 canbe performed by the various embodiments of the image-based particlesorting system 100 in accordance with the present technology, such asthe system 200 and the system 1100.

The method 400 includes a process 405 to capture image data, by theimaging system 120, of a particle flowing through a channel in theparticle flow device 110, e.g., at the interrogation area 115. Forexample, the process 405 can include continuously capturing images at apredetermined rate or varying rates, which may be based on the particleflow speed in the channel. In some implementations, the process 405includes receiving, at a controller of the imaging system 120, an imagecapture command from the data processing and control unit 230 to affectthe image capture protocol to obtain the image data. For example, thedata processing and control unit 130 can change one or more parametersof the image capture protocol executed by the imaging system 120 inreal-time implementations of the system.

The image capture rate can be associated with the data volume of singleparticles. For example, the system can determine the image capture rateand/or other parameters of image capture protocol based at least in parton (a) the particle flow speed of the particle flow device 110 and (b)the electronic sampling rate, e.g., depending on what resolution isdesired, which can be used to determine the data volume of singleparticles. The image capture rate can be associated with the datarecording/computing capabilities of the data processing and control unit130 and/or a controller (e.g., processor) of the imaging system 120. Forexample, higher speed analog-to-digital conversion (ADC) and largermemory can increase the image capture rate. The image capture protocolincluding image capture and processing parameters can be selected basedat least in part on that the design of the spatial filter, e.g., as insome implementations the optical output from the spatial filter canaffect the processing of the algorithm complexity. Notably, opticalfactors, such as magnification, optical filters, imaging mode, etc.typically have little, if any, influence on the image capture speed, butare significant in obtaining the input data that is processed to producethe overall results, e.g., determination of the particle properties thatare evaluated for determining a sorting decision or other analyses.

The method 400 includes a process 410 to receive the image data, at thedata processing and control unit 130 and from the imaging system 120, inwhich the received data is associated with a particle imaged by theimaging system 120 flowing in the channel of the particle flow device110. For example, in some implementations of the process 410, the dataprocessing and control unit 130 receives time domain signal data, e.g.,optical intensity, from one or more PMTs of the imaging system 120 foreach particle imaged in the illumination area on the particle flowdevice 110. The method 400 includes a process 420 to process the data(e.g., image signal data) to produce an image data set representative ofan image of the particle imaged by the imaging system 120. For example,in some implementations of the process 420, the received image signaldata is pre-processed by filtering the data, reconstructing an imagebased on the filtered data using an image reconstruction algorithm inaccordance with the present technology, and/or resizing thereconstructed image, e.g., in which the reconstructed image is convertedto binary image data. The method 400 includes a process 430 to analyzethe produced image data set to identify one or more features of theimaged particle based on predetermined criteria and to determine asorting command based on the one or more identified features. In someimplementations of the method, the method 400 includes a process 415 tofirst process the received image signal data to detect a presence of theparticle in the illumination area, and then process the data inaccordance with the process 420 to produce the image data set that issubsequently analyzed in accordance with the process 430. The method 400includes a process 440 to provide the sorting command to the actuator140. The method 400 includes a process 445 to execute, by the actuator140, the sorting command to direct the particle flowing in sorting areaof the channel of the particle flow device 110 to the correspondingoutput channel of the device 110. For example, in implementations of themethod 400, the imaging system 120 and the data processing and controlsystem 230 implement the processes 405 and 410, 420, 430, and 440,respectively, in real time, such that the actuator 140 receives andexecutes the sorting command to direct the particles accordingly, e.g.,gate each particle to the appropriate output channel of the device 110,within a diminutive time period from the image capture (process 405) tothe actuation of particle gating (process 445). For example, in someimplementations of the process 445, the actuator 140 includes apiezoelectric actuator coupled to the flow device 110 to producehydrodynamic deflection in the fluid that causes the particle to move ina particle direction along a desired trajectory to enter the desiredoutput channel.

In some embodiments, the data processing and control unit 130 includessoftware modules corresponding to one or any combination of theprocesses 410, 420, 430 and 440 stored in the memory 132 and executableby the processor 131 to implement the processes 410, 420, 430 and/or440. FIG. 4B shows a diagram of example data processing modules of thesystems in accordance with the present technology. For example, the dataprocessing and control unit 130 can include a Particle Detection module461, a Process Image module 463, and a Make Sorting Decision module 465.As shown by the example diagram of FIG. 4B, once a particle (e.g., cell)is detected by implementing the algorithms executable by the ParticleDetection module 461, the data processing and control unit 130 proceedsto implement the Process Image module 463 if a particle (e.g., cell) isdetected, otherwise, it returns to keep recording images, e.g., via PMTreadout. By extracting parameters from cell images and comparing thosevalues to pre-defining sorting criteria, e.g., performed by the ProcessImage module 463, the Make Sorting Decision module 465 is implemented todetermine whether to trigger the actuator 140 or not. In someembodiments, the data processing and control unit 130 includes a RecordImage module 467 to control, at least partially, the imaging system 120to implement the process 405, e.g., which can include initiating and/oradapting the image capture protocol or settings of the imaging system.For example, in some implementations such as with the imager 222, whenthe Record Image module 467 is operable to control the recording of thePMT readout in a certain length.

In some implementations, the parameters associated with particleproperties (e.g., cell properties) are extracted based on the type ofparameter or parameters to be extracted. For example, for differentmorphology parameters, the process to extract the parameters associatedwith cell morphological properties can include at least some of thefollowing techniques. The process can include analyzing the image area,for example, including determining the number of pixels with value “1”in the binary image. The number of pixels is image area. The process caninclude analyzing the perimeter, for example, including determining thenumber of pixels on the image contour, e.g., as the image contour isdetected. The process can include analyzing the diameter in the xdirection, for example, including (e.g., in a binary image) determiningthe number of “1” pixels in each row that represents the x direction.For example, the largest count can be determined as the diameter in xdirection. The process can include analyzing the diameter in the ydirection, for example, including (e.g., in a binary image), determiningthe number of “1” pixels in each column that represents the y direction.For example, the largest count can be determined as the diameter in ydirection.

FIG. 4C shows a diagram of an example implementation of the process 430,e.g., implementation of the Particle Detection module 461, to detectcells for image-based cell sorting. In some implementations, forexample, the Particle Detection module 461 searches for a section of thetime-domain PMT signal that has integrated fluorescence intensity,referred to herein as “brightness”, that is larger than a presetthreshold. For example, in some implementations, the cell-travellingspeed in the flow cell or microfluidic device is 0.08 m/s, and a 200 kHzsampling rate is used, such that the particle sorting and imageprocessing algorithm integrates every consecutive 500 data points toobtain the value of brightness. Every time when the brightness is largerthan a first preset threshold, e.g., threshold1, this means a cell isentering the optical system's field of view in the image plane. Thealgorithm determines the time derivative of brightness, which is thencompared to a second preset threshold, e.g., threshold2. If the timederivative is smaller than threshold2, the algorithm considers that thecell is well within the field of view, and the example process 430continues to determine an identified feature or features of the detectedcell, e.g., implementation of the Process Image module 463 isconsequently initiated.

FIG. 4D shows a brightness-time plot depicting an example cell detectionprocess implementation based on the processing of a time-domain PMTsignal associated with a single cell traveling through the illuminationarea of the system. As shown in the plot, a cell is detected to enterthe imaging area (e.g., interrogation area 115) based on a threshold(e.g., brightness >1.0) just after the 1.176 s time marker, and thesystem captures image signal data until the cell is detected to departthe imaging area based on the threshold (e.g., brightness <1.0) afterthe 1.179 s time marker.

FIG. 5A shows a diagram of an example implementation of the process 430,e.g., implementation of the Process Image module 463, to determine anidentified feature or features of the detected cell for image-based cellsorting. In the example shown in FIG. 5A, the process 420 is implementedafter some aspects of the process 430, e.g., particularly after analysisof image signal data to determine if a cell is detected or not. Forexample, in some implementations as shown in FIG. 5A, after the partialanalysis of the received image signal data to determine the presence ofa cell, the Process Image module 463 pre-processes the image signal data(e.g., time-domain PMT signal) including filtering or other signalprocessing techniques. For example, in some implementations, thereceived image signal data is low-pass filtered to eliminate highfrequency noise. In some examples, a 10^(th) order hamming window forlow-pass filtering is used, which is an example of a particular low passfilter applicable. The Process Image module 463 can implement otherfeatures of the process 420 including image reconstruction techniques(e.g., based on an example algorithm described herein, includingEquation (1)) to turn the time-domain signal into an image thatrepresents 2-dimensional spatial distribution of the cells'fluorescence. After that, the reconstructed cell image is resized forfidelity purpose and converted into binary image. For binary images, forexample, an optional open filter technique is applied to remove spuriousnoise, and the Process Image module 463 detects one or more cellfeatures (e.g., the cell wall or membrane detection) after the openfilter is applied. In some implementations, the open filter can beapplied after cell feature(s) have been detected. The Process Imagemodule 463 extracts parameters based on the detected cell features thatdescribes aspects of the cell, such as cell morphology, which can becompared to the values from user's preset sorting criteria forsubsequent determination of the sorting command (e.g., gating command).

Example implementations of the method 400 using the system 200 aredescribed below for an example study demonstrating image-based cellsorting. In the example study, the imaging system 220 included a 100 mW488-nm laser (e.g., iBeam-SMART, Toptica) that has an oval beam shapewith Gaussian energy distribution, which is collimated, focused, andthen expanded to illuminates an area of 100 μm (x-direction) by 250 μm(y-direction) to form the illumination area 215 on the microfluidicdevice 210. The fluorescence passing the miniature dichroic mirror with500 nm cutoff wavelength (e.g., ThorLabs) and the scattering light werecollected through a 50×, 0.55NA objective lens (e.g., Mituyoyo). Thelight intensity signal in each channel was acquired by a PMT (e.g.,H9307-02, Hamamatsu).

Image reconstruction techniques, implemented by the Process Image module463, for example, were used for spatial-to-temporal transformation ofimage data that can be mathematically formulated in the following:

S(t)=∫_(x,y)Cell(x,y−Mvt)·F(x,y)·I(x,y)dxdy  (1)

where S(t) is the measured PMT signal, Cell is the two-dimensional cellfluorescence or scattering intensity profile, F(x, y) is thecharacteristic function of the spatial filter, I(x, y) is the intensityprofile of laser illumination, y is along the cell-travelling directionand x is along the transverse direction, and M is the magnificationfactor of the optical system pertaining to the flow cytometer. As thecell travels in the microfluidic channel at a speed v, the imageprojected onto the spatial filter, e.g., the example SF 224 shown inFIG. 2B, travels at an effective speed of Mv. In the simplest case toexplain the principle and solving for Cell in Equation (1), one canchoose F(x,y) to be a series of small slits (e.g., 100 μm by 50 μmrectangular slits) represented approximately in Equation (2) and I(x, y)to be a constant from a laser beam of uniform intensity (i.e., top-hatbeam profile):

F(x,y)=Σ_(q=1) ^(N)δ(x−q)·δ(y−qL)  (2)

where x=1, 2, . . . , N is the number of row in the spatial filter, L isthe distance between two slits that transmits fluorescence. As a result,for example, the cell image can be constructed from the followingrelation:

$\begin{matrix}{{{Cell}\left( {x,y} \right)} = {S\left( \frac{{xL} - y}{v} \right)}} & (3)\end{matrix}$

An example image reconstruction technique that can be implemented by theProcess Image module 463, for example, includes determining twovariables: the number of sampling points in each peak, and the startingpoint of the time-domain image signal data (e.g., PMT signal) for thecell flowing. For example, since cells are not travelling at perfectlyuniform speed, number of sampling points in each peak slightly changes;and since cell travelling speed as well as cell position in the imagearea slightly change (e.g., 20 μm by 20 μm image area set based on theexample spatial filter), the starting point of the time-domain PMTreadout also changes. In the example image reconstruction technique, mis referred to starting point of PMT readout, n is referred to number ofpoints in each peak. Based on cell speed variation, for example, nranges from 46 to 51. Accordingly, m ranges from 0 to 519-10n. Theexample image reconstruction technique sweeps m and n to assure the bestcombination to reconstruct cell image. Summation of intensities atstarting point of each peak is calculated for every combination in thesweeping, for example. The combination with smallest summation is theright answer for image reconstruction. After m and n are calculated,they can be used to reconstruct both bright field and fluorescenceimages since signal recorded by both PMTs are synchronized. The exampleimage reconstruction technique calculates values based on Equation (4),

$\begin{matrix}{\left( {m,n} \right) = \left\{ \left( {m,n} \right) \middle| {\min {\sum\limits_{i = 0}^{10}\; {\sum\limits_{m = 0}^{519 - {10\; n}}\; {{FIFO}_{{PMT}\; 1}\left\lbrack {m + {i \times n}} \right\rbrack}}}} \right\}} & (4)\end{matrix}$

FIG. 5B shows an example data plot depicting the results of animplementation of the image reconstruction technique, e.g., which showsexample results of the search for the best combination of starting pointand number of sampling points for each peak. The black “*” symbols shownin the data plot represent starting points of each peak found by thereconstruction algorithm.

Generally, with the spatial filter described above (e.g., spatial filter224 of FIG. 2B) inserted at the image plane, fluorescence from differentparts of the cell will pass different slits at different times. As aresult, the waveform of the fluorescent signal from the PMT includes asequence of patterns separated in time domain, and each section of thesignal in the time domain corresponds to the fluorescent signalgenerated by each particular regime of the cell. After the lightintensity profile over each slit is received, the cell image of theentire cell can be constructed by splicing all the profile together. Inthe example embodiment of the system 200 shown in FIGS. 2A-2C, thespatial filter contains ten 100 μm by 50 μm rectangular slits positionedin sequence. With a 50× objective lens (M=50), for example, the filterdesign allows construction of the fluorescent or scattering image of atravelling cell, e.g., no larger than 20 μm by 20 using the algorithmincluding the Equation (3), which requires a minimum amount ofcomputations and is suitable for high-throughput, real-time image-basedcell classification and sorting. For example, using a 50×/0.55NAobjective lens, 500 kHz sampling rate for acquiring PMT signal, and 0.2m/s cell-travelling speed that is given by 12 μL/min sample flow rateand 120 μL/min sheath flow rate, the effective size of the pixel iny-direction is

$\frac{L}{\left( {L/{Mv}} \right) \times R} = {\frac{Mv}{R} = {0.4\mspace{14mu} {\mu m}}}$

which is smaller than the Rayleigh Criterion, thus resulting in adiffraction-limited resolution in y-direction. Here R is the samplingrate of PMT readout in this calculation. It is noted that otherimplementations parameters can be used, such as: cell-travelling atspeed around 8 cm/s (e.g., given by the flow rate); image area set to be20 μm by 20 μm (e.g., PMT readout based on fluorescence image of 10peaks); sampling rate of 200 kSamples/s, such that each peak includes 50sampling points.

For the example study, the design of spatial filter was drawn in AutoCADand printed to a transparency mask at 20,000 dots per inch (dpi). Alayer of negative photoresist (e.g., NR9-1500PY, Futurrex, Inc.) wasspun at 3,000 rotations per minute (rpm) on a 6-inch glass wafer. Thewafer was heated on a hot plate at 150° C. for 3 minutes then exposed toUV light (e.g., EVG620NT, EV Group) through the transparency mask. PostUV exposure, the wafer was baked at 100° C. for another 3 minutes beforedevelopment in RD6 (e.g., Futurrex, Inc.) for 12 seconds. A film of 200nm thick aluminum was sputtered onto the glass wafer. After metallift-off, the patterns of the spatial filter were formed and the glasswafer was diced into 15 mm by 15 mm pieces. To help hold the spatialfilter in the flow cytometer system, the spatial filter having ten 50 μmby 100 μm slits was mounted to a sample holder fabricated by 3D printingmethod.

For the example study, HEK293T human embryonic kidney cell samples weretransfected with pEGFP-GR plasmids (e.g., Addgene). After continuousculturing for 3 days, 1 μM dexamethasone (e.g., Sigma-Aldrich) was addedto the culture media. After incubation for 60 minutes, the HEK293T cellswere harvested, fixed by 4% paraformaldehyde, washed and resuspended in1× phosphate buffered saline (PBS). Before every imaging experiment, thesuspension was diluted in PBS to a concentration of 200 cells/μL.

The example embodiment of the image-based cell sorting system 200 usedin the example study included a spatial mask including 10 slits andutilized a cell flow speed of around 0.08 m/s. The image area was set tobe 20 μm by 20 In this design, the PMT signal from a cell's fluorescenceappears to be 10 separated peaks, and number of sampling points of eachpeak is around 50. However, the speed of different cells is notperfectly uniform, so the number of sampling points for each cellslightly changes, and the starting point of each peak also needs to bedetermined. As such, the data processing algorithms were configured toaccount for such varying cell speeds, i.e., to search for the startpoint and number of sampling points for each peak within a certainrange, so that the cell image can be successfully reconstructed. Basedon the variations in cell flowing speed, the number of sampling pointsof each peak typically ranged from 46 to 51. So the total number ofpoints in the cell image is from 460 to 510. For each cell, once theParticle Detection module 461 determined that there comes a cell, thePMT signal in a length of 520 sampling points was recorded. For example,to assure the best combination of staring point and number of samplingpoints for each peak, the algorithm sweeped the number of samplingpoints from 46 to 51 and start points of each peaks accordingly. Foreach combination in the sweeping, the sum of intensities at all startingpoints was calculated, and the combination with smallest summation waschosen as correct answer for image reconstruction.

FIG. 6. shows a data plot depicting the fluorescence intensity signalfor a cell based on an example image reconstruction. The example resultare indicative of searching for the best combination of staring pointand number of sampling points for each peak.

FIG. 7 shows an example of a reconstructed image of a cell, e.g., inwhich the criteria was used the determine boundaries of a cell usingfluorescent area to produce the image of the cell. For the open filter,which is an erosion followed by a dilation, a 3 by 3 neighborhood wasused. In the step of cell wall or membrane detection, all the pixels ina binary cell image are scanned. For the pixels that have non-zerointensity, the example algorithm checked all nine pixels in its 3 by 3neighborhood. By counting its neighboring pixels that have non-zerointensities, for example, if the counted number is larger than 0 andsmaller than 8, this pixel was determined as a pixel on cell wall ormembrane.

The example technique was implemented in at least in-part using an FPGA.Table 1 shows the round-up approximate latency results for each step inthe example implementation of the data processing algorithm, in whichthe total latency is well within 3.3 ms. The example data processingalgorithm is flexible to be implemented on multiple platforms, includingin parallel. For example, utilizing the parallel processing power ofgraphics processing unit (GPU), the algorithm can also be implemented inGPU, by, for example, either the CUDA architecture by Nvidia or theOpenCL by AMD. The image soring algorithm has a much shorter runtimecombining the parallel processing power of GPU, so the sortingthroughput is further improved. Although the formulation of producingcell images is described in the form of example methods in this patentdocument, it is worth noticing that the algorithm can somewhat differwhen using different spatial filter designs than the previouslydescribed 10-slit spatial mask. The overall working procedure of thesystem, however, remains the same.

Table 1 shows example data depicting the performance of modules in theexample FPGA design, e.g., in which latencies is represented in time(ms).

TABLE 1 Latency (ms) Low-pass filter 0.6 Reconstruct image 0.4 Resizeimage 0.8 Open filter 0.9 Detect cell wall 0.4 Extract morphologyparameter(e.g., cell area) 0.2 Total 3.3

Example sorting results from the example study are described. Forexample, to demonstrate the feasibility of the example real-timeimage-based cell sorter system, sorting tests on a mixture of cellspossessing fluorescent cytoplasm and cells possessing fluorescentnucleus were performed. The normal human embryonic kidney cells, HEK293Tcells, after transfection with pEGFP-GR plasmids, are expressing GFPthat can be excited by 488 nm laser and has an emission peak at 509 nmin their cytoplasm. After 1 μM dexamethasone treatment for 1 hour, thefluorescence translocates to the cell nucleus from the cytoplasm region.

FIGS. 8A and 8B show example fluorescence microscope images oftransfected but not drug treated cells (FIG. 8A) and transfected anddrug treated cells (FIG. 8B). The images of FIGS. 8A and 8B represent anarea of 20 μm by 20 μm. As shown by the example representativemicroscopic images of the transfected HEK293T cells without drugtreatment (FIG. 8A), and those of treated cells (FIG. 8B), the drugtreated cell image has smaller fluorescence area, even though themagnitude of the fluorescence intensity of both treated and untreatedcells has a quite wide range.

As shown in the images of FIG. 8A and FIG. 8B, the left column of bothrows shows the fluorescence image; the middle column shows the brightfield image; and the right column shows the overlay images. Row 1 showsthe sample cell with fluorescence distributed in cytoplasm (e.g.,untreated with dexamethasone). Row 2 shows the sample cell withfluorescence distributed in nucleus (treated with dexamethasone). Thecells are pEGFP-GR plasmids transfected HEK293T human embroynic kidneycells.

FIGS. 9A and 9B show fluorescence cell images taken by the image-basedcell sorter system of transfected but not drug treated cells (FIG. 9A)and transfected and drug treated cells (FIG. 9B). The example imagesrepresent an area of 20 μm by 20 μm.

FIG. 10 shows a histogram of the example calculated fluorescence area ofall events and of the sorted cells from the example study.

In the example study, the sorting criteria was preset to be 84 μm², sothe drug treated cells are sorted by the example system. In this examplestudy, due to the variation in cell size, nucleus size, and drug uptakeby cells, it is possible that not all drug treated cells were sorted,but, the sorted cells all have fluorescence only in nucleus, so thepurity is secured.

As demonstrated in the example implementation of an example embodimentof the system 200, the disclosed systems and techniques provide ahigh-throughput flow cytometer with cell sorting capabilities based onsingle cells' fluorescent and light scattering images. Realizing thecell image capture, process and sorting actuation in FPGA and/or GPU,the example results show that the afore-described design provides anoverall processing latency at millisecond level.

FIG. 11 shows a diagram of an image-based cell sorting microfluidicsystem 1100 in accordance with some embodiments of the image-basedparticle sorting system 100. The system 1100 includes a microfluidicdevice 1110 to flow particles through an optical interrogation channelfor sorting, an imaging system 1120 to obtain image data of theparticles in an illumination area of the interrogation channel, a dataprocessing and control system 1130 to process the obtained image data inreal time and determine a sorting command, and an actuator 1140 to gatethe particles in the microfluidic device 210 based on the determinedsorting command. In some implementations of the example system 1110, themicrofluidic device 1110 and the actuator 1140 can include themicrofluidic device 210 and the actuator 240, respectively.

The imaging system 1120 of the system 1100 includes a light source 1121,e.g., a laser, to provide an input or probe light at the illuminationarea of the microfluidic device 1110, and an optical imager 1122 toobtain images of the illuminated particles in the illumination area 215.The example optical imager 1122, as shown in FIG. 11, includes anobjective lens 1123 (e.g., of a microscope or other optical imagingdevice) optically coupled to a spatial filter (SF) 1124, emissionfilters (EF) 1125A and 1125B, and photomultiplier tubes (PMT) 1126. Inthe example implementation shown in FIG. 11, the imaging system 1120includes dichroic mirrors (DM) 1129A and 1129B, in which DM 1129A isarranged with the light source to direct the input light at theillumination area on the microfluidic device 1110 and DM 1129B isarranged with the optical imager 1122 in the optical path to direct aportion of the optical output signal to the PMT 1126B via the EF 1125B,while the undirected portion of the optical output signal proceeds toPMT 1126A via the EF 1125A. In some implementations, the light source1121 (e.g., the laser) is configured to produce a fluorescent excitationsignal that is incident upon the illumination area to cause afluorescent emission by the particles. The optical imager 1122 capturesthe optical output fluorescent emission signal at the PMTs 1126A and1126B, such that an image of the particle can be generated.

In example implementations of the system 1100, suspended single cellsare hydrodynamically focused in the sorting channel of the microfluidicdevice 1110 by sheath flow, ensuring that the cells travel in the centerof the fluidic channel at a uniform velocity. For example, bothfluorescence emission and bright field signal can be detected by themultiple photomultiplier tubes, e.g., PMTs 1126A and 1126B. Toaccommodate the geometry of the microfluidic device, a 488 nm laser beamfrom the laser 1121 is introduced to the optical interrogation by52-degree reflection by a miniature dichroic mirror 1129A positioned infront of a 50× objective lens (e.g., NA=0.55, working distance=13 mm).In some example embodiments, the system 1100 includes an optical lightsource 1128 (e.g., LED, such as the example 405 nm LED shown in FIG.11), which can be placed at the opposite side of the channel to generatebright field images, and in which the light can be focused at the laserillumination position. The spatially coded filter, e.g., SF 1124, isinserted at the image plane in the detection path. To route desiredemission band to their respective PMTs, the dichroic mirror 1129B splitsthe fluorescent and bright field light collected by the objective lensbased on spectrum. The spatial resolution of the reconstructed image isdetermined by the filter. Resolution in x- (transverse) directiondepends on the number of slits on the filter, and in y-(cell-travelling) direction depends on the sampling rate and cell flowspeed. In the example embodiment of the system 1100 shown in FIG. 11,the effective pixel size is 2 μm in x-direction and about 0.4 μm iny-direction.

FIG. 12 shows a flow diagram of an example image processingimplementation of an example embodiment of the system 1100. The dataprocessing and control unit 1130 includes image processing modules toprocess fluorescence and bright field images captured by the imagingsystem 1120. An example image processing module, e.g., to implementaspects of the processes 420 and 430 of the method 400, for example,includes pre-processing the received image signal data from the multiplePMTs 1126A and 1126B. In such implementations, the pre-processingtechniques can be performed after a detection of a cell is determinedbased on analysis of the received image signal data. In the exampleshown in FIG. 12, the PMT signals of fluorescence and bright fieldimages are low-pass filtered to eliminate high frequency noise. In someimplementations, for example, a 10th order hamming window is used forlow-pass filtering. The example image processing module is configured toexecute an image reconstruction algorithm to reconstruct both brightfield and fluorescence image from time-domain PMT signal, e.g., into 2dimensional images. Since bright field and fluorescence signals aregenerated by the same slits, they are synchronized by the dataprocessing and control unit 1130. In some implementations, the imagereconstruction algorithm is launched once for both bright field andfluorescence images. In some implementations, for example, reconstructedimages are resized to 50×50 pixels. In some implementations, forexample, grayscale images are converted to binary image based onintensity threshold. In some implementations, for example, binary imagesare filtered by open filter to eliminate spurious noise. In someimplementations, for example, cell contour algorithm is launched todetect contour of both binary images. In some implementations, forexample, morphology parameters are extracted based on processed imagesand sorting decision is made based on extracted parameters.

FIG. 13A shows a flow diagram depicting example data through the exampleimage processing steps shown in FIG. 12. In the diagram, the exampleimage processing module produces a reconstructed and resized image ofthe detected individual cell including image contour features, whichallow for cellular feature parameters such as cell morphology to beextracted from the produced image.

The algorithm is implemented on FPGA (National Instrument cRIO-9039).Table 2 shows example round-up approximate latency results for each stepimplemented by the example image processing module of the dataprocessing and control unit 1130. Image processing for bright field andfluorescence images is executed in parallel. As shown in the table, thetotal latency of image processing is around 5.8 ms for this example.

Table 2 shows example data depicting the performance of modules in theexample FPGA design for example implementations of the system 1100,e.g., in which latencies is represented in time (ms).

TABLE 2 Latency (ms) Low-pass filter 0.8 Reconstruction 0.4 Resize 1.6Open filter 2 Cell contour detection 1 Total 5.8

By decreasing the time latency of image processing, for example, thesorting throughput can be improved. With more powerful FPGA that hasmore computational resource, for example, the image processing modulecan be further paralleled to improve time latency. The example imageprocessing algorithm can also be implemented on a GPU alternatively oradditionally (e.g., in parallel processing with FPGA), e.g., such as byeither the CUDA architecture by Nvidia or the OpenCL by AMD as examples.Because the example image processing algorithm is data-parallel, byutilizing the parallel processing power of GPU, the processing speed canbe much accelerated, so that a much higher throughput can be achieved.

In some implementations, for example, the data processing and controlunit 1130 can be configured to evaluate extracted morphology parameters,e.g., using Receiver Operating Characteristics (ROC). The top parametersare selected for real-time sorting.

In some implementations, a method for evaluating extracted morphologyparameters using ROC technique includes the following. For example, theROC technique can be applied to processed data from the image dataobtained by flow cell samples through the system. For example, the cellimages can be separated into two subsets, e.g., in some instances bymanual identification. As an example, the two subsets can betranslocated cells (e.g., a sorted group) and un-translocated cells(e.g., an unsorted group). For each cell, morphology parameters areextracted. The ROC technique can include generating a distribution foreach parameter of both subsets. The technique includes generating thecurve, which is the ROC curve for the distribution. The techniqueincludes calculating the area under the curve (AUC), i.e., theintegration of the curve. The technique includes evaluating how theparameters apply for the classification, e.g., the parameter with thelarger AUC can be selected as the best suitable parameter forclassification.

FIG. 13B shows an example distribution plot for subsets using an exampleROC technique to evaluate extracted parameters such as cell morphologyparameters. In the example shown, the ROC includes TP, FP, FN, TN (TP istrue positive, FP is false positive, FN is false negative, TN is truenegative). The example curve represents the ROC curve. The parameterwith larger AUC is better for the classification.

After parameters are selected for sorting, Support Vector Machine (SVM)can be used to generate nonlinear hyperplane as gating criterion usingthe selected parameters.

As an example, after the images are separated into subsets, e.g., suchas the two example subsets, the selected parameters are calculated foreach image. Each image corresponds to a n dimensional vector (n is thenumber of selected parameters). The example SVM technique is implementedto generate the boundary in n dimensional space that separates thesubsets. For example, in making sorting decision, the selectedparameters for each cell can be calculated, e.g., so as to know the cellis on which side of the boundary (e.g., which subsets does the cellbelong to). Then, the sorting decision can be made.

Example extracted morphology parameters are shown in Table 3.

TABLE 3 Fluorescence Bright field Fluorescence image + image imageBright field image Area Area Fluorescence area/bright field area (Arearatio) Perimeter Perimeter Fluorescence perimeter/bright field perimeter(Perimeter ratio) Shape factor Shape factor (Area/Perimeter)(Area/Perimeter) Diameter Diameter (in x direction) (in x direction)Diameter Diameter (in y direction) (in y direction)

Example implementations of the method 400 using the system 1100 aredescribed below for example studies demonstrating image-based cellsorting, including sorting based on protein translocation, sorting basedon cell life cycle, and sorting based on number of beads bonded on thecell membrane of cells.

One example study included the sorting of pEGFP-GR plasmids translocatedHEK-297T Human embryonic kidney cells, e.g., an implementation ofsorting based on protein translocation. The example study demonstratedthe capability to identify and sort pEGFP-GR plasmids translocatedHEK-297T Human embryonic kidney cells from translocated andun-translocated mixtures. In the study, HEK-293T Human embryonic kidneycells were transfected with GR-GFP and separated into 2 plates. Oneplate of cells was untreated so the fluorescence stays in cytoplasm. Theother plate of cells was treated with drug so the fluorescence migratesfrom cytoplasm to nucleus. Both types of cells were mixed and flowthrough the system, in which only translocated cells were sorted andcollected based on implementation of the example method.

In the example study, the recorded PMT signal was processed by the dataprocessing and control unit, e.g., executing the algorithms implementedby Matlab code. Based on the PMT signal, both fluorescence and brightfield images of each cell were reconstructed, and morphology parametersof each cell were extracted and recorded. The extracted morphologyparameters were used for supervised machine learning to generatecriteria for real-time image-based cell sorting.

In the example study, the reconstructed cell images were separated intotwo subsets by manual identification. One subset of cells wasun-translocated cells with fluorescence in cytoplasm, the other subsetof cells was translocated cells with fluorescence in nucleus.

FIGS. 14A and 14B show cell images of un-translocated cells andtranslocated cells captured by the example system and reconstructed bythe Matlab code. FIG. 14A shows images of un-translocated cells, andFIG. 14B shows images of translocated cells. In the images of FIGS. 14Aand 14B, the image on the left is the fluorescence image of the cell,the image in the middle is bright field image of the cell, and the imageon the right is overlay image with detected bright field contour.

in the example study, morphology parameters were evaluated based on thetwo annotated subsets using Receiver Operating Characteristics (ROC).The top 3 parameters were selected for real-time sorting. For example,the top 3 parameters in this case were Fluorescence area/Bright fieldarea (e.g., area ratio), Fluorescence perimeter/Bright field perimeter(e.g., perimeter ratio), and Fluorescence area.

In the example study, sorting criteria using the three selectedparameters was employed for real-time sorting of the cells in thesystem. A three dimensional nonlinear hyperplane separating the two cellsubsets based on the selected top three parameters was formed by SupportVector Machine (SVM).

FIG. 15 shows an example of a hyperplane exhibiting separation of thetwo cell sets (e.g., translocated cells and un-translocated cells) forimplementations of the sorting criteria.

The two types of cells were mixed 50:50 and diluted to 200/μL usingphosphate buffered saline (PBS). The mixed sample were flowed throughthe example image-based sorting system and sorted based on the real-timemodule. The cells traveled at a speed of 8 cm/s that is given by 6μL/min sample flow rate and 60 μL/min sheath flow rate. The exampleimager included a microscope with a CCD camera that was used to captureboth fluorescence and bright field images of collected cells.

FIGS. 16A and 16B show cell images of un-translocated cells andtranslocated cells captured by the example system, via the microscope.FIG. 16A shows the microscope images of un-translocated cells, and FIG.16B shows the microscope images of translocated cells. In the images ofFIGS. 16A and 16B, the image on the left is the fluorescence image ofthe cell imaged by the microscope, the image in the middle is brightfield image of the cell by the microscope, and the image on the right isoverlay image.

One example study included the sorting of MDCK Madin-Darby Canine KidneyEpithelial cells at G2/M stage, e.g., an implementation of sorting basedon cell life cycle. In the study, MDCK Madin-Darby Canine KidneyEpithelial cells were fixed and cell nucleus was stained with PropidiumIodide (PI). Fixed and stained MDCK cells were flowed through theexample image-based sorting system, and only cells at G2/M phase weresorted and collected.

Similar to the example study for sorting based on protein translocation,the recorded PMT signals were processed to reconstruct cell images andextract morphology parameters. The reconstructed cell images wereseparated into two subsets by manual identification. One subset of cellswas at G1 phase. At G1 phase, constituents of nucleus are confined innucleus membrane. The other subset of cells was at G2/M phase. At G2/Mphase, nucleus membrane breaks down and constituents of nucleus aredistributed within the cell.

FIGS. 17A and 17B show cell images of G1 phase cells and G2/M phasecells captured by the example system and reconstructed by the Matlabcode. FIG. 17A shows images of cells at G1 phase, and FIG. 17B showsimages of cells at G2/M phase. In the images of FIGS. 17A and 17B, theimage on the left is the fluorescence image of the cell, the image inthe middle is bright field image of the cell, and the image on the rightis overlay image with detected bright field contour.

Similar to the example study for sorting based on protein translocation,morphology parameters were evaluated using ROC, and a three dimensionalnonlinear hyperplane was formed by SVM. The top 3 morphology parametersare Fluorescence area/Bright field area (e.g., area ratio), Fluorescencearea and Fluorescence perimeter/Bright field perimeter (e.g., perimeterratio).

FIG. 18 shows an example of a hyperplane exhibiting separation of thetwo cell sets (e.g., cells at G1 phase and cells at G2/M) forimplementations of the sorting criteria. MDCK cells were diluted to200/μL using PBS. The sample was flowed through the example system andsorted based on the real-time module. The cells traveled at speed 8 cm/sthat is given by 6 μL/min sample flow rate and 60 μL/min sheath flowrate. The example imager included a microscope with a CCD camera thatwas used to capture both fluorescence and bright field images ofcollected cells.

FIGS. 19A and 19B show cell images of G1 phase cells and G2/M cellscaptured by the example system, via the microscope. FIG. 19A shows themicroscope images of cells at phase G1, and FIG. 19B shows themicroscope images of cells at G2/M phase. In the images of FIGS. 19A and19B, the image on the left is the fluorescence image of the cell imagedby the microscope, the image in the middle is bright field image of thecell by the microscope, and the image on the right is overlay image.

One example study included the sorting of HEK-297T Human embryonickidney cells based on number of beads bond with the cells, e.g., animplementation of sorting based on number of beads bonded on the cellmembrane of cells. In the study, HEK-2971 Human embryonic kidney cellswere bonded with fluorescence beads and stained with CFSE kits. Thefluorescence of cells was at 520 nm and fluorescence of beads is at 645nm. Fluorescence signals at two wavelengths at routed by dichroic mirrorand detected by two PMTs. Cells were sorted based on number of bondedbeads. The image processing module for this application was modifiedaccordingly to process the images.

FIG. 20 shows a flow diagram of an example image processingimplementation for sorting based on number of beads bonded on the cellmembrane of cells implemented by an example embodiment of the system1100. The data processing and control unit 1130 includes imageprocessing modules to process fluorescence and bright field imagescaptured by the imaging system 1120. An example image processing module,e.g., to implement aspects of the processes 420 and 430 of the method400, for example, includes pre-processing the received image signal datafrom the multiple PMTs 1126A and 1126B. As shown in the diagram, forexample, the fluorescence signals at both wavelengths were low-passfiltered. The fluorescence signal at 520 nm was used for imagereconstruction in this example study. The reconstructed beads image wasresized to 50×50 pixels. The top-hat transform was implemented to removeimage background. In this example implementation, a 7×7 pixelsneighborhood was used for top-hat transform. In this exampleimplementation, a grayscale image was converted to binary image.Morphology parameters were extracted. In this example, image area waschosen as sorting criterion as beads have relatively uniform size. Insome implementations, for example, morphology parameters are extractedbased on processed images and sorting decision is made based onextracted parameters.

Table 4 shows the total latency of real-time image processing is 7.8 ms.The example modules were implemented in using FPGA for beads counting.

TABLE 4 Latency (ms) Low-pass filter 0.8 Reconstruction 0.4 Resize 1.6Top-hat transform 5 Total 7.8

FIG. 21 shows examples of grayscale cell images processed by the imageprocessing module with different number of beads. As shown in FIG. 21,the images in the left column are fluorescence images of beads, theimages in the middle column are fluorescence image of the cells, and theimages in the right column are overlay images.

FIG. 22 shows a histogram of beads image area for different number ofbeads.

EXAMPLES

The following examples are illustrative of several embodiments inaccordance with the present technology. Other exemplary embodiments ofthe present technology may be presented prior to the following listedexamples, or after the following listed examples.

In some embodiments in accordance with the present technology (exampleA1), an image-based particle sorting system includes a particle flowdevice structured to include a substrate, a channel formed on thesubstrate operable to flow particles along a flow direction to a firstregion of the channel, and two or more output paths branching from thechannel at a second region proximate to the first region in the channel;an imaging system interfaced with the particle flow device and operableto obtain image data associated with a particle when the particle is inthe first region during flow through the channel; a data processing andcontrol unit in communication with the imaging system, the dataprocessing and control unit including a processor configured to processthe image data obtained by the imaging system to determine one or moreproperties associated with the particle from the processed image dataand to produce a control command based on a comparison of the determinedone or more properties with a sorting criteria; and an actuatoroperatively coupled to the particle flow device and in communicationwith the actuator, the actuator operable to direct the particle into anoutput path of the two or more output paths based on to the controlcommand, in which the system is operable to sort each of the particlesduring flow in the channel within a time frame of 15 ms or less from afirst time of image capture by the imaging system to a second time ofparticle direction by the actuator.

Example A2 includes the system of example A1, in which the particle flowdevice includes a microfluidic device or a flow cell integrated with theactuator on the substrate of the microfluidic device or the flow cell.

Example A3 includes the system of example A1, in which the actuatorincludes a piezoelectric actuator coupled to the substrate and operableto produce a deflection to cause the particle to move in a direction inthe second region that directs the particle along a trajectory to theoutput path of the two or more output paths.

Example A4 includes the system of example A1, in which the imagingsystem includes one or more light sources to provide an input light atthe first region of the particle flow device, and an optical imager tocapture the image data from the particles illuminated by the input lightin the first region.

Example A5 includes the system of example A4, in which the one or morelight sources include at least one of a laser or a light emitting diode(LED).

Example A6 includes the system of example A4, in which the opticalimager includes an objective lens of a microscope optically coupled to aspatial filter, an emission filter, and a photomultiplier tube.

Example A7 includes the system of example A6, in which the opticalimager further includes one or more light guide elements to direct theinput light at the first region, to direct light emitted or scattered bythe particle to an optical element of the optical imager, or both.

Example A8 includes the system of example A7, in which the light guideelement includes a dichroic mirror.

Example A9 includes the system of example A6, in which the opticalimager includes two or more photomultiplier tubes to generate two ormore corresponding signals based on two or more bands or types of lightemitted or scattered by the particle.

Example A10 includes the system of example A1, in which the processor ofthe data processing and control unit includes a field-programmablegate-array (FPGA), a graphics processing unit (GPU), or a FPGA and a GPUin parallel.

Example A11 includes the system of example A1, in which the dataprocessing and control unit is configured to receive the image data(e.g., including time domain signal data) associated with the particleimaged in the first region on the particle flow device, process theimage data to produce an image data set representative of an image ofthe particle, analyze the produced image data set to extract one or moreparameters from the image data set associated with the one or moreproperties of the particle, and determine the control command based onthe comparison of the extracted one or more parameters with one or morethresholds of the sorting criteria.

Example A12 includes the system of example A11, in which the dataprocessing and control unit is configured to process the image data toproduce the image data set by filtering the image data, reconstructing afirst image based on the filtered data, and resizing the reconstructedfirst image to produce a second image, in which the second imageincludes binary image data.

Example A13 includes the system of example A1, in which the particlesinclude cells, and the one or more properties associated with the cellincludes an amount or a size of a features of or on the cell, one ormore sub-particles attached to the cell, or a particular morphology ofthe cell or portion of the cell.

Example A14 includes the system of example A1, in which the particlesinclude cells, and the sorting criteria includes a cell contour, a cellsize, a cell shape, a nucleus size, a nucleus shape, a fluorescentpattern, or a fluorescent color distribution.

Example A15 includes the system of example A1, in which the particlesinclude cells, and the one or more properties associated with the cellincludes a physiological property of the cell including a cell lifecycle phase, an expression or localization of a protein by the cell, adamage to the cell, or an engulfment of a substance or sub-particle bythe cell.

In some embodiments in accordance with the present technology (exampleA16), a method for image-based sorting of a particle includes obtainingimage signal data of a particle flowing through a channel of a particleflow device; processing the image signal data to produce an image dataset representative of an image of the particle; analyzing the producedimage data set to identify one or more properties of the particle fromthe processed image data; producing a control command by evaluating theone or more identified properties with a sorting criteria; and directingthe particle into one of a plurality of output paths of the particleflow device based on to the control command.

Example A17 includes the method of example A16, in which the obtaining,the processing, the analyzing, the producing and the directingoperations are performed during the particle's flow in the channelwithin a time frame of 15 ms or less.

Example A18 includes the method of example A16, in which the producingthe control command includes extracting one or more parameters from theprocessed image data of the particle associated with the identified oneor more properties of the particle, and comparing the extracted one ormore parameters from the image with one or more threshold values of thesorting criteria.

Example A19 includes the method of example A16, in which the processingthe image signal data to produce the image data set includes filteringthe image signal data; reconstructing a first image based on thefiltered data; and resizing the reconstructed first image to produce asecond image, in which the second image includes binary image data.

Example A20 includes the method of example A16, in which the processingthe image signal data includes detecting the presence of the particleprior to producing the image data set representative of the image of theparticle.

Example A21 includes the method of example A20, in which the detectingthe presence of the particle includes calculating a brightness valueassociated with a magnitude of signal intensity of the image signaldata; evaluating that the brightness value with a first threshold;determining a derivative value when the brightness value exceeds thefirst threshold; evaluating that the derivative value exceeds a secondthreshold; determining that the derivative value exceeds the secondthreshold.

Example A22 includes the method of example A16, in which the particleflowing through the channel is a cell, and the one or more propertiesassociated with the cell includes an amount or a size of a features ofor on the cell, one or more sub-particles attached to the cell, or aparticular morphology of the cell or portion of the cell.

Example A23 includes the method of example A16, in which the particleflowing through the channel is a cell, and the sorting criteria includesa cell contour, a cell size, a cell shape, a nucleus size, a nucleusshape, a fluorescent pattern, or a fluorescent color distribution.

Example A24 includes the method of example A16, in which the particleflowing through the channel is a cell, and the one or more propertiesassociated with the cell includes a physiological property of the cellincluding a cell life cycle phase, an expression or localization of aprotein by the cell, a damage to the cell, or an engulfment of asubstance or sub-particle by the cell.

In some embodiments in accordance with the present technology (exampleA25), a method for obtaining image based sorting of cells in a flowcytometry device includes recording an image; analyzing the recordedimage to determine whether or not a cell is detected; when adetermination that a cell is detected is determined, processing theimage; and based on the results of the processing, making adetermination whether to trigger an actuator in the flow cytometrydevice.

Example A26 includes the method of example A25, in which upon adetermination that a cell is not detected, resuming the recording of theimage.

Example A27 includes the method of example A25, in which determiningwhether or not a cell is detected includes calculating a brightnessvalue; determining whether or not the brightness value exceeds a firstthreshold; when a determination that the brightness value exceeds thefirst threshold is made, determining a derivative value; determiningwhether or not the derivative value exceeds a second threshold; and whena determination that the derivative value exceeds the second thresholdis made, providing an indication that a cell is detected, in which whenno determination that the brightness value exceeds the first thresholdor the derivative value exceeds the second threshold, providing anindication that a cell is not detected or providing no indication.

In some embodiments in accordance with the present technology (exampleA28), a particle imaging and flow system includes a particle flow devicestructured to include a substrate, a channel formed on the substrateoperable to flow particles along a flow direction to a first region ofthe channel; an imaging system interfaced with the particle flow deviceand operable to obtain image data associated with a particle when theparticle is in the first region during flow through the channel; and adata processing and control unit in communication with the imagingsystem, the data processing and control unit configured to process theimage data obtained by the imaging system and to determine one or moreproperties associated with the particle to produce an analyzed data setincluding data indicative of the determined one or more properties ofthe particle.

Example A29 includes the system of example A28, in which the particleflow device includes a microfluidic device or a flow cell.

Example A30 includes the system of example A28, in which the imagingsystem includes one or more light sources to provide an input light atthe first region of the particle flow device, and an optical imager tocapture the image data from the particles illuminated by the input lightin the first region.

Example A31 includes the system of example A30, in which the one or morelight sources include at least one of a laser or a light emitting diode(LED).

Example A32 includes the system of example A30, in which the opticalimager includes an objective lens of a microscope optically coupled to aspatial filter, an emission filter, and a photomultiplier tube.

Example A33 includes the system of example A32, in which the opticalimager further includes one or more light guide elements (e.g., dichroicmirror) to direct the input light at the first region, to direct lightemitted or scattered by the particle to an optical element of theoptical imager, or both.

Example A34 includes the system of example A30, in which the opticalimager includes two or more photomultiplier tubes to generate two ormore corresponding signals based on two or more bands or types of lightemitted or scattered by the particle.

Example A35 includes the system of example A28, in which the processorof the data processing and control unit includes a field-programmablegate-array (FPGA), a graphics processing unit (GPU), or a FPGA and a GPUin parallel.

Example A36 includes the system of example A28, in which the dataprocessing and control unit is configured to receive the image data(e.g., including time domain signal data) associated with the particleimaged in the first region on the particle flow device, process theimage data to produce an image data set representative of an image ofthe particle, and analyze the produced image data set to extract one ormore parameters from the image data set associated with the one or moreproperties of the particle.

Example A37 includes the system of example A36, in which the dataprocessing and control unit is configured to process the image data toproduce the image data set by filtering the image data, reconstructing afirst image based on the filtered data, and resizing the reconstructedfirst image to produce a second image, in which the second imageincludes binary image data.

Example A38 includes the system of example A28, in which the particlesinclude cells, and the one or more properties associated with the cellincludes an amount or a size of a features of or on the cell, one ormore sub-particles attached to the cell, or a particular morphology ofthe cell or portion of the cell.

Example A39 includes the system of example A28, in which the particlesinclude cells, and the sorting criteria includes a cell contour, a cellsize, a cell shape, a nucleus size, a nucleus shape, a fluorescentpattern, or a fluorescent color distribution.

Example A40 includes the system of example A28, in which the particlesinclude cells, and the one or more properties associated with the cellincludes a physiological property of the cell including a cell lifecycle phase, an expression or localization of a protein by the cell, adamage to the cell, or an engulfment of a substance or sub-particle bythe cell.

In some embodiments in accordance with the present technology (exampleB1), a system includes a microfluidic device integrated with on-chipsorting actuator; a high-speed and high-sensitivity imaging opticalsystem; and a real-time cell image processing and sorting controlelectronic system.

Example B2 includes the system of example B 1, in which the a real-timecell image processing and sorting control electronic system isconfigured to allow one or more user-defined gating criteria.

Example B3 includes the system of example B1, in which the one or moreuser-defined gating criteria includes a cell contour, a cell size, acell shape, a size and shape of nucleus, a fluorescent pattern, or afluorescent color distribution.

In some embodiments in accordance with the present technology (exampleB4), a method for obtaining image based sorting of cells in a flowcytometry device includes recording an image; determining whether or nota cell is detected, and upon a determination that a cell is detected,processing the image; and based on the results of the processing, makinga determination whether to trigger an actuator in the flow cytometrydevice.

Example B5 includes the method of example B4, in which upon adetermination that a cell is not detected, resuming the recording of theimage.

Example B6 includes the method of example B4, in which determiningwhether or not a cell is detected includes calculating a brightnessvalue, upon a determination that the brightness value exceed a firstthreshold, determining a derivative value, and upon a determination thatthe derivative value exceed a second threshold, providing an indicationthat a cell is detected.

Example B7 includes the method of example B4, which is implemented inone of an FPGA or a DSP.

Example B8 includes the system of example B1, in which the real-timecell image processing and sorting control electronic system isimplemented in an FPGA.

In some embodiments in accordance with the present technology (exampleC1), an image-based particle sorting system includes a particle flowdevice structured to include a substrate, a channel formed on thesubstrate operable to flow cells along a flow direction to a firstregion of the channel, and two or more output paths branching from thechannel at a second region proximate to the first region in the channel;an imaging system interfaced with the particle flow device and operableto obtain image data associated with a cell when the cell is in thefirst region during flow through the channel; a data processing andcontrol unit in communication with the imaging system, the dataprocessing and control unit including a processor configured to processthe image data obtained by the imaging system to determine one or moreproperties associated with the cell from the processed image data and toproduce a control command based on a comparison of the determined one ormore properties with a sorting criteria, in which the control command isproduced during the cell flowing in the channel and is indicative of asorting decision determined based on one or more cellular attributesascertained from the image signal data that corresponds to the cell; andan actuator operatively coupled to the particle flow device and incommunication with the actuator, the actuator operable to direct thecell into an output path of the two or more output paths based on to thecontrol command, in which the system is operable to sort each of thecells during flow in the channel within a time frame of 15 ms or lessfrom a first time of image capture by the imaging system to a secondtime of particle direction by the actuator.

Example C2 includes the system of example C1, in which the particle flowdevice includes a microfluidic device or a flow cell integrated with theactuator on the substrate of the microfluidic device or the flow cell.

Example C3 includes the system of example C1, in which the actuatorincludes a piezoelectric actuator coupled to the substrate and operableto produce a deflection to cause the cell to move in a direction in thesecond region that directs the cell along a trajectory to the outputpath of the two or more output paths.

Example C4 includes the system of example C1, in which the imagingsystem includes one or more light sources to provide an input light atthe first region of the particle flow device, and an optical imager tocapture the image data from the cells illuminated by the input light inthe first region.

Example C5 includes the system of example C4, in which the one or morelight sources include at least one of a laser or a light emitting diode(LED).

Example C6 includes the system of example C4, in which the opticalimager includes an objective lens of a microscope optically coupled to aspatial filter, an emission filter, and a photomultiplier tube.

Example C7 includes the system of example C6, in which the opticalimager further includes one or more light guide elements to direct theinput light at the first region, to direct light emitted or scattered bythe cell to an optical element of the optical imager, or both.

Example C8 includes the system of example C7, in which the light guideelement includes a dichroic mirror.

Example C9 includes the system of example C6, in which the opticalimager includes two or more photomultiplier tubes to generate two ormore corresponding signals based on two or more bands or types of lightemitted or scattered by the cell.

Example C10 includes the system of example C1, in which the processor ofthe data processing and control unit includes a field-programmablegate-array (FPGA), a graphics processing unit (GPU), or a FPGA and a GPUin parallel.

Example C11 includes the system of example C1, in which the dataprocessing and control unit is configured to receive the image dataincluding time domain signal data associated with the cell imaged in thefirst region on the particle flow device, process the image data toproduce an image data set representative of an image of the cell,analyze the produced image data set to extract one or more parametersfrom the image data set associated with the one or more propertiesassociated with the cell, and determine the control command based on thecomparison of the extracted one or more parameters with one or morethresholds of the sorting criteria.

Example C12 includes the system of example C11, in which the dataprocessing and control unit is configured to process the image data toproduce the image data set by filtering the image data, reconstructing afirst image based on the filtered data, and resizing the reconstructedfirst image to produce a second image, in which the second imageincludes binary image data.

Example C13 includes the system of example C1, in which the one or moreproperties associated with the cell includes one or more of an amount ora size of a feature of or on the cell, one or more particles attached tothe cell, or a particular morphology of the cell or portion of the cell.

Example C14 includes the system of example C1, in which the sortingcriteria includes a cell contour, a cell size, a cell shape, a nucleussize, a nucleus shape, a fluorescent pattern, or a fluorescent colordistribution.

Example C15 includes the system of example C1, in which the one or moreproperties associated with the cell includes a physiological property ofthe cell including a cell life cycle phase, an expression orlocalization of a protein by the cell, an expression or localization ofa gene by the cell, a damage to the cell, or an engulfment of asubstance or a particle by the cell.

Example C16 includes the system of example C15, in which the determineddamage to the cell includes DNA damage.

In some embodiments in accordance with the present technology (exampleC17), a method for image-based particle sorting includes obtaining imagesignal data of a cell flowing through a channel of a particle flowdevice; processing the image signal data to produce an image data setrepresentative of an image of the cell; analyzing the produced imagedata set to identify one or more properties of the cell from theprocessed image data; evaluating the one or more identified propertiesof the cell with a sorting criteria to produce a control command to sortthe cell based on one or more cellular attributes ascertained from theimage signal data corresponding to the cell during cell flow in theparticle flow device; and directing the cell into one of a plurality ofoutput paths of the particle flow device based on to the controlcommand.

Example C18 includes the method of example C17, in which the obtaining,the processing, the analyzing, the producing and the directingoperations are performed during the cell flowing in the channel within atime frame of 15 ms or less.

Example C19 includes the method of example C17, in which the producingthe control command includes extracting one or more parameters from theprocessed image data of the cell associated with the identified one ormore properties of the cell, and comparing the extracted one or moreparameters from the image with one or more threshold values of thesorting criteria.

Example C20 includes the method of example C17, in which the processingthe image signal data to produce the image data set includes filteringthe image signal data; reconstructing a first image based on thefiltered data; and resizing the reconstructed first image to produce asecond image, in which the second image includes binary image data.

Example C21 includes the method of example C17, in which the processingthe image signal data includes detecting the presence of the cell priorto producing the image data set representative of the image of the cell.

Example C22 includes the method of example C21, in which the detectingthe presence of the cell includes calculating a brightness valueassociated with a magnitude of signal intensity of the image signaldata; evaluating that the brightness value with a first threshold;determining a derivative value when the brightness value exceeds thefirst threshold; evaluating that the derivative value exceeds a secondthreshold; and determining that the derivative value exceeds the secondthreshold.

Example C23 includes the method of example C17, in which the one or moreproperties associated with the cell includes one or more of an amount ora size of a features of or on the cell, one or more particles attachedto the cell, or a particular morphology of the cell or portion of thecell.

Example C24 includes the method of example C17, in which the sortingcriteria includes a cell contour, a cell size, a cell shape, a nucleussize, a nucleus shape, a fluorescent pattern, or a fluorescent colordistribution.

Example C25 includes the method of example C17, in which the one or moreproperties associated with the cell includes a physiological property ofthe cell including a cell life cycle phase, an expression orlocalization of a protein by the cell, an expression or localization ofa gene by the cell, a damage to the cell, or an engulfment of asubstance or a particle by the cell.

Example C26 includes the method of example C25, in which the determineddamage to the cell includes DNA damage.

Implementations of the subject matter and the functional operationsdescribed in this patent document can be implemented in various systems,digital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.Implementations of the subject matter described in this specificationcan be implemented as one or more computer program products, i.e., oneor more modules of computer program instructions encoded on a tangibleand non-transitory computer readable medium for execution by, or tocontrol the operation of, data processing apparatus. The computerreadable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “data processing unit” or “dataprocessing apparatus” encompasses all apparatus, devices, and machinesfor processing data, including by way of example a programmableprocessor, a computer, or multiple processors or computers. Theapparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Computer readable media suitable for storingcomputer program instructions and data include all forms of nonvolatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

It is intended that the specification, together with the drawings, beconsidered exemplary only, where exemplary means an example. As usedherein, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. Additionally, the use of “or” is intended to include“and/or”, unless the context clearly indicates otherwise.

While this patent document contains many specifics, these should not beconstrued as limitations on the scope of any invention or of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments of particular inventions. Certain features thatare described in this patent document in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in this patent document should not be understoodas requiring such separation in all embodiments.

Various embodiments described herein are described in the generalcontext of methods or processes, which may be implemented in oneembodiment by a computer program product, embodied in acomputer-readable medium, including computer-executable instructions,such as program code, executed by computers in networked environments. Acomputer-readable medium may include removable and non-removable storagedevices including, but not limited to, Read Only Memory (ROM), RandomAccess Memory (RAM), compact discs (CDs), digital versatile discs (DVD),Blu-ray Discs, etc. Therefore, the computer-readable media described inthe present application include non-transitory storage media. Generally,program modules may include routines, programs, objects, components,data structures, etc. that perform particular tasks or implementparticular abstract data types. Computer-executable instructions,associated data structures, and program modules represent examples ofprogram code for executing steps of the methods disclosed herein. Theparticular sequence of such executable instructions or associated datastructures represents examples of corresponding acts for implementingthe functions described in such steps or processes.

For example, one aspect of the disclosed embodiments relates to acomputer program product that is embodied on a non-transitory computerreadable medium. The computer program product includes program code forcarrying out any one or and/or all of the operations of the disclosedembodiments.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this patent document.

1. An image-based particle sorting system, comprising: a particle flowdevice structured to include a substrate, a channel formed on thesubstrate operable to flow cells along a flow direction to a firstregion of the channel, and two or more output paths branching from thechannel at a second region proximate to the first region in the channel;an imaging system interfaced with the particle flow device and operableto obtain image data associated with a cell when the cell is in thefirst region during flow through the channel; a data processing andcontrol unit in communication with the imaging system, the dataprocessing and control unit including a processor configured to processthe image data obtained by the imaging system to determine one or moreproperties associated with the cell from the processed image data and toproduce a control command based on a comparison of the determined one ormore properties with a sorting criteria, wherein the control command isproduced during the cell flowing in the channel and is indicative of asorting decision determined based on one or more cellular attributesascertained from the image signal data that corresponds to the cell; andan actuator operatively coupled to the particle flow device and incommunication with the actuator, the actuator operable to direct thecell into an output path of the two or more output paths based on to thecontrol command, wherein the system is operable to sort each of thecells during flow in the channel within a time frame of 15 ms or lessfrom a first time of image capture by the imaging system to a secondtime of particle direction by the actuator.
 2. The system of claim 1,wherein the particle flow device includes a microfluidic device or aflow cell integrated with the actuator on the substrate of themicrofluidic device or the flow cell.
 3. The system of claim 1, whereinthe actuator includes a piezoelectric actuator coupled to the substrateand operable to produce a deflection to cause the cell to move in adirection in the second region that directs the cell along a trajectoryto the output path of the two or more output paths.
 4. The system ofclaim 1, wherein the imaging system includes one or more light sourcesto provide an input light at the first region of the particle flowdevice, and an optical imager to capture the image data from the cellsilluminated by the input light in the first region.
 5. The system ofclaim 4, wherein the one or more light sources include at least one of alaser or a light emitting diode (LED).
 6. The system of claim 4, whereinthe optical imager includes an objective lens of a microscope opticallycoupled to a spatial filter, an emission filter, and a photomultipliertube.
 7. The system of claim 6, wherein the optical imager furtherincludes one or more light guide elements to direct the input light atthe first region, to direct light emitted or scattered by the cell to anoptical element of the optical imager, or both.
 8. The system of claim7, wherein the light guide element includes a dichroic mirror.
 9. Thesystem of claim 6, wherein the optical imager includes two or morephotomultiplier tubes to generate two or more corresponding signalsbased on two or more bands or types of light emitted or scattered by thecell.
 10. The system of claim 1, wherein the processor of the dataprocessing and control unit includes a field-programmable gate-array(FPGA), a graphics processing unit (GPU), or a FPGA and a GPU inparallel.
 11. The system of claim 1, wherein the data processing andcontrol unit is configured to receive the image data including timedomain signal data associated with the cell imaged in the first regionon the particle flow device, process the image data to produce an imagedata set representative of an image of the cell, analyze the producedimage data set to extract one or more parameters from the image data setassociated with the one or more properties associated with the cell, anddetermine the control command based on the comparison of the extractedone or more parameters with one or more thresholds of the sortingcriteria.
 12. The system of claim 11, wherein the data processing andcontrol unit is configured to process the image data to produce theimage data set by filtering the image data, reconstructing a first imagebased on the filtered data, and resizing the reconstructed first imageto produce a second image, wherein the second image includes binaryimage data.
 13. The system of claim 1, wherein the one or moreproperties associated with the cell includes one or more of an amount ora size of a feature of or on the cell, one or more particles attached tothe cell, or a particular morphology of the cell or portion of the cell.14. The system of claim 1, wherein the sorting criteria includes a cellcontour, a cell size, a cell shape, a nucleus size, a nucleus shape, afluorescent pattern, or a fluorescent color distribution.
 15. The systemof claim 1, wherein the one or more properties associated with the cellincludes a physiological property of the cell including a cell lifecycle phase, an expression or localization of a protein by the cell, anexpression or localization of a gene by the cell, a damage to the cellincluding DNA damage, or an engulfment of a substance or a particle bythe cell.
 16. (canceled)
 17. A method for image-based particle sorting,comprising: obtaining image signal data of a cell flowing through achannel of a particle flow device; processing the image signal data toproduce an image data set representative of an image of the cell;analyzing the produced image data set to identify one or more propertiesof the cell from the processed image data; evaluating the one or moreidentified properties of the cell with a sorting criteria to produce acontrol command to sort the cell based on one or more cellularattributes ascertained from the image signal data corresponding to thecell during cell flow in the particle flow device; and directing thecell into one of a plurality of output paths of the particle flow devicebased on to the control command.
 18. The method of claim 17, wherein theobtaining, the processing, the analyzing, the producing and thedirecting operations are performed during the cell flowing in thechannel within a time frame of 15 ms or less.
 19. The method of claim17, wherein the producing the control command includes extracting one ormore parameters from the processed image data of the cell associatedwith the identified one or more properties of the cell, and comparingthe extracted one or more parameters from the image with one or morethreshold values of the sorting criteria.
 20. The method of claim 17,wherein the processing the image signal data to produce the image dataset includes: filtering the image signal data; reconstructing a firstimage based on the filtered data; and resizing the reconstructed firstimage to produce a second image, wherein the second image includesbinary image data.
 21. The method of claim 17, wherein the processingthe image signal data includes detecting the presence of the cell priorto producing the image data set representative of the image of the cell.22. The method of claim 21, wherein the detecting the presence of thecell includes: calculating a brightness value associated with amagnitude of signal intensity of the image signal data; evaluating thatthe brightness value with a first threshold; determining a derivativevalue when the brightness value exceeds the first threshold; evaluatingthat the derivative value exceeds a second threshold; and determiningthat the derivative value exceeds the second threshold.
 23. The methodof claim 17, wherein the one or more properties associated with the cellincludes one or more of an amount or a size of a features of or on thecell, one or more particles attached to the cell, or a particularmorphology of the cell or portion of the cell.
 24. The method of claim17, wherein the sorting criteria includes a cell contour, a cell size, acell shape, a nucleus size, a nucleus shape, a fluorescent pattern, or afluorescent color distribution.
 25. The method of claim 17, wherein theone or more properties associated with the cell includes a physiologicalproperty of the cell including a cell life cycle phase, an expression orlocalization of a protein by the cell, an expression or localization ofa gene by the cell, a damage to the cell including DNA damage, or anengulfment of a substance or a particle by the cell.
 26. (canceled)